System and Process for Mobile Object Tracking

ABSTRACT

Embodiments include system and processes for tracking objects using a camera. An optical marker dictionary including one or more optical markers is generated, the optical markers being optically distinct indicators. An optical marker within the optical marker dictionary is associated with and affixed to an object. A processor is in communication with the camera, receiving image data from the camera and applying computer vision to the image data in order to detect the presence of one or more optical markers within the optical marker dictionary within the image data. The processor determines camera position information and applies computer vision to the image data in order to determine relative position information for the detected optical markers and projects a position from the camera to a detected optical marker.

BACKGROUND Field of the Invention

The present invention relates to object tracking, more specifically toobject tracking using optical systems.

Description of the Related Art

It can be helpful to track a distinct, moving object or objects overtime using a camera.

SUMMARY

Embodiments of the present invention are directed to system andprocesses for tracking objects using an imager. An optical markerdictionary including one or more optical markers is generated, theoptical markers being optically distinct indicators. An optical markerwithin the optical marker dictionary is associated with and affixed toan object. A processor is in communication with the imager, receivingimage data from the imager. The processor applies computer vision to theimage data in order to detect the presence of one or more opticalmarkers within the optical marker dictionary within the image data. Theprocessor determines camera position information and applies computervision to the image data in order to determine relative positioninformation for the detected optical markers. Employing the cameraposition information and the detected optical markers relative positionas input, the processor projects a position from the camera to adetected optical marker.

These and other features, aspects, and advantages of the invention willbecome better understood with reference to the following description,and accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a diagram of major components of a system for anembodiment of the current invention as it may exist in operation;

FIGS. 2A-2C depict a diagram of major components of a system for anembodiment of the current invention as it may exist in operation;

FIG. 3 depicts representative optical markers and optical markerdictionaries;

FIG. 4 depicts a flowchart for an embodiment of a process of the currentinvention;

FIGS. 5A and 5B depict a diagram of major components of a system for anembodiment of the current invention as it may exist in operation; and

FIG. 6 depicts representative communication of tracked objects.

DETAILED DESCRIPTION

Detailed descriptions of the preferred embodiment are provided herein.It is to be understood, however, that the present invention may beembodied in various forms. Therefore, specific details disclosed hereinare not to be interpreted as limiting, but rather as a basis for theclaims and as a representative basis for teaching one skilled in the artto employ the present invention in virtually any appropriately detailedsystem, structure or manner.

Embodiments of the present invention are directed to systems andprocesses for object tracking using an imager to detect optical markersaffixed to those objects. FIG. 1 illustrates an embodiment of systemsaccording to the present invention. Depicted are objects 12 (O1, O2),with affixed optical markers 14, within the field of view 26 of one ormore cameras 20, the cameras 20 in communication with a computer 40.

Objects 12 include mobile items, such as people. For example, persons 12might be a police officer or soldier in an emergency scenario in anurban theatre. In other cases, a person 12 might be a child in themepark.

Optical markers 14 are optically distinct indicators to be displayed onan object 12 and associated with that particular object 12 to whichcomputer vision will be applied to detect and classify the opticalmarker 14. FIG. 3 illustrates example optical markers 14 with unique,optically detectable, distinguishable characteristics. Exemplary opticalmarkers 14 are those which are detected and classified with minimalprocessing time, false positives, and false negatives for an environmentand camera. Representative distinct characteristics includes shapes andspectrum. Representative suitable optical marker 14 shapes include, butare not limited to, alphanumeric or symbols. In certain configurations,suitable optical markers 14 include letters, numbers, or symbols of alanguage such as 0-9, a-z, and A-Z. For example, an optical marker 14can include symbols such as a circle, square, triangle, star, wrench,gear, or other shapes. In certain configurations, machine-readableoptical label formats such as bar codes, QR codes, or the like areincluded, where the encoded content differs, providing distinct opticalmarkers. Representative suitable spectrum include the visible spectrumand others such as ultraviolet, infrared, or others known in the art tobe detectable by camera 20 sensors. In certain configurations, anoptical marker 14 is generated with known, target dimensions, such as alength and width. In certain configurations, the optical marker 14 canbe configured to be temporary detectable. For example, the opticalmarker 14 may fade over a period of time. In certain configurations, oneor more optical characteristics are combined for an optical marker 14,such as a shape having a particular color. An optical marker dictionary19 includes multiple, distinct optical markers 14.

An optical marker 14 is affixed to and associated with the object 12.FIG. 2B illustrates an optical marker 14 affixed to a person 12. Theoptical marker 14 is affixed to an exposed area of the object 12, suchas the torso or helmet. Certain configurations include an adhesive labelhaving a print surface one side and an adhesive surface. In suchconfigurations, a printer having a suitable inks, toners, or othersubstrate is used to imprint the optical marker 14 on the print surface.For example, a printer with print media to match the desired opticalcharacteristics for the desired optical marker 14 is used to imprint thedesired optical marker 14 to the print surface. The printed labelbearing the optical marker 14 can then be affixed to the person 12. Inother configurations, the optical marker 14 is incorporated in fabric,where it can be incorporated into garments such as a shirts, vests,jackets, wraps, helmet covers, or the like. In such configurations,fabric using suitable thread, dyes, screens, or other substrate orsubstrate applicator is used to incorporate the optical marker 14 withthe fabric. Certain configurations include an electronic visual displaysuch as liquid crystal displays, electroluminescent displays, LEDdisplays, quantum dot displays, electronic paper displays, and the like.In such configurations, lighting of the appropriate spectrum is used todisplay the optical marker 14.

An imager 20, such as a camera 20 is a device for recording visualimages in the forms of image data such as still or video signals. Asuitable camera 20 is one which detects the corresponding wavelength(s)of the optical marker 14, such as the common visual spectrum,fluorescent, infrared, ultraviolet, or other spectrum using filters,sensors, post-processing or other means known in the art. In certainconfigurations, one or more cameras 20 is deployed and oriented to atarget area, typically above it, operable to monitor for optical markers14 within the field(s) of view 26. For example, cameras 20 might bewithin a unmanned aerial vehicle 22 or street light assembly. One ormore cameras 20 may form a network for multiple fields of view 26 or anextended field of view 26, stitching the fields of view in certainconfigurations.

The cameras 20 are in communication with a computer 40, the computeroperable to process image data from the camera(s) 20. A computer, asreferred to in this specification, generally refers to a system whichincludes a processor, memory, a screen, a network interface, andinput/output (I/O) components connected by way of a data bus. The I/Ocomponents may include for example, a mouse, keyboard, buttons, or atouchscreen. An exemplary computer is a portable computer such as ahandheld computer, smartphone, or tablet computer, wearable (eg glasses,watches), such as an iOS device, Android based device, or other similarcomputer. The portable computer is optionally configured with a touchscreen and integrated camera elements. Those skilled in the art willappreciate that the computer can take a variety of configurations,including personal computers, hand-held devices, multi-processorsystems, microprocessor-based electronics, network PCs, minicomputers,mainframe computers, and the like. Additionally, the computer may bepart of a distributed computer environment where tasks are performed bylocal and remote processing devices that are communicatively linked.Although shown as separate devices, one skilled in the art canunderstand that the structure of and functionality associated with theaforementioned elements can be optionally partially or completelyincorporated within one or the other, such as within one or moreprocessors. The computers 40 are communicatively coupled with the one ormore cameras 20 over wired or wireless connection.

FIG. 4 illustrates an embodiment of a process of the current invention.At step 110, optical markers are assigned to the objects. At step 120,optical markers are affixed to the objects. At step 130, cameras aremonitored. At step 140, image data is received from the cameras. At step150, camera position is determined. At step 160, camera data isprocessed for optical marker detection. At step 170, optical markerpositions are determined. At step 180, position information iscommunicated. More consideration of each of the steps is given below.

At step 110, optical markers are assigned to the objects. At step 110,one or more optical markers 14 is created and/or selected for thescenario. FIG. 2A generally illustrates a sample environment for a givenscenario, a building interior with large items that can be barriers topersons 12 being directly visible to one another. A unique opticalmarker 14 is selected for each object 12 to be monitored. Factors inselecting an optical marker set 19 include selecting optical markers 14that can be reliably detected in the target environment and that havelow probability for detection collision with another optical marker 14in the set. For example, a green colored optical marker 14 would likelybe avoided in a forest target area and selecting oval and circle opticalmarker 14 within the same optical marker dictionary 19 would likely beavoided. In exemplary configuration, the assigned objects 12 areassociated with an object identifier. For instance, a circle opticalmarker 14 might be associated with an “O1” object identifier. Furtherinformation may be associated with the object identifier, such as skillsor properties of the object 12.

At step 120, optical markers 14 are affixed to the object(s) to anexposed area, such as a head or body. In exemplary configuration, aunique optical marker 14 is affixed to each object 12. In otherconfigurations, a unique optical marker 14 is affixed to classes orcategories of objects 12. For instance, where the target medium for theoptical marker 14 is an adhesive label, the optical marker 14 isimprinted on the print surface and the label placed on an exposed partof the object 12. For instance, where the target medium for the opticalmarker 14 is a fabric, the optical marker 14 is sewn, dyed, screened, orotherwise incorporated into the fabric and the fabric is donned on theobject 12. For instance, where the target medium for the optical marker14 is an electronic visual display, the optical marker 14 image data istransmitted to the electronic visual display and the electronic visualdisplay is secured to the object 12. FIG. 2B generally illustratesoptical markers 14 associated and affixed to objects 12.

At step 130, cameras 20 are monitored. Cameras 20 and objects 12 aredeployed or otherwise move about an area. For example, unmanned aerialvehicles 22 with cameras 20 or street lights with cameras 20 may bedeployed. One or more cameras 20 are in communication with one or morecomputers 40. Communication of the image data from the camera(s) 20 isenabled and set up. For instance, the camera 20 of a unmanned aerialvehicle 22 communicate setup can include enabling a radio, such as an802.11 connection, or a tether. For instance, the camera 20 of a streetlight may include a network card for wide area network communication.

At step 140, image data from cameras 140 is received by the computer. Incertain configurations, the optical characteristics about the image orcamera(s) 20 are received, such as the focal length field width, zoomcapabilities, current zoom, current focus, and other information.

At step 150, the camera 20 position is determined. In certainconfigurations, the position is periodically updated, with positioninformation having associated timestamps. It is within the scope of theinvention to employ absolute, relative, or hybrid positioning. Inconfigurations that employ absolute positioning, position informationfrom global navigation systems can be employed. For instance, affixedwith the camera 20 are receivers for systems such as the GlobalPositioning System, Galileo, GLONASS, Wi-Fi positioning systems, indoorpositioning systems, or the like. Position information received fromabsolute positions systems is commonly received as latitude, longitude,and altitude information.

In configurations that employ relative positioning, position informationfrom an origin point can be employed. Suitable origin points can includean earlier position of the camera 20 (such as the launch position of theunmanned aerial vehicle 22 to which the camera 20 is affixed or theknown position of the street light to which the camera 20 is affixed),anchor nodes (nodes with known fixed position), or other means known inthe art. Over time, position information is updated if the camera 20position changes. For example, speeds and directions of travel overtime, time of flight/time of arrival signal processing from a referencedevice with a known location, and other means known in the art can beemployed to determine position relative to an origin point. FIGS. 1 and2C illustrate representative display of camera 20 position.

At step 160, the camera 20 image data is processed for optical markers14. Periodically, the computer 40 receives image data from the camera 20in order to detect the presence of optical markers 14. FIG. 2A shows asample image from which image data in the field of view 26 of a camera20 is generated. A computer 40 is used to process image data fromcameras 20 using computer vision in order to detect the presence of oneor more optical markers 14 within the optical markers dictionary 19.

In exemplary configuration, a processor is in communication with thecamera 20 in order to receive and process the image data from a camera20. A processor is included in a computer. The processor applies imageclassification approaches to detect the presence of optical markers 14in the image data received from the camera 20. Machine learning, such asneural networks, may be employed. Some employed aspects of computervision of the current embodiments include object detection andrecognition, shape detection, blob analysis, position analysis, motiontracking, motion estimation, thresholding, optical characterrecognition, optical decoding (eg machine-readable optical label formatssuch as QR codes), and other aspects. Representative object detectionapproaches include Viola Jones object detection, Single Shot MultiBoxDetector, You Only Look Once, neural network based, Faster RCNN, andothers known in the art. To illustrate, computer vision can be appliedto the image data from the camera 20 to detect optical markers 14,determine distance from the camera 20, determine position within theimage data, determine relative position within the image data, and otherimage processing. One or more optical markers 14 may be detected.

At step 170, position information for detected optical markers 14 isdetermined. FIGS. 2C, 5A, and 5B illustrate aspects of determiningposition information for detected optical markers 14. If one or moreoptical markers 14 are detected, its position and its correspondingassigned person 12 are determined. A position relative to the camera 20for each of the detected optical markers 14 is determined. In exemplaryconfiguration, computer vision is employed to determine relativepositions. In certain configurations, the distance and angle of anoptical marker 14 relative to the camera 20 is employed to determined arelative position. Representative information employed for computervision based distance and angle determination includes optical marker 14dimensions, camera information (such as lens, focal point, field width,position of the camera, orientation of the camera . . . ), camerasettings for an image (such as zoom settings, field width, . . . ),optical marker 14 position in the image frame, optical marker 14 size inthe image frame, changes in optical marker 14 position or dimensionsacross image frames, reference objects, image resolution, other opticalmarker 14 position information, and other information. For example,approaches such as stereo photography, triangle similarity, motionanalysis, trajectory-based projections, and other means known in the artcan be employed for computer vision based distance determination of theoptical marker 14. Other configurations employ laser range finder,radar, lidar, ultrasonic, and other means known in the art for distancedetermination.

For example, approaches such as position in image data and other meansknown in the art can be employed for computer vision based determinationof the angle orientation relative to the camera 20. Other configurationsemploy global navigation system data (eg bearing), compasses, motionanalysis, trajectory-based projections, and other means known in the artfor angular determination.

Information such as the distance and relative angle provide relativeposition information for the optical marker 14. Further information maysupplement relative position determination such as altitude information.Using the relative position information of the optical marker 14, aprojection from the camera 20 position to determine position informationfor that optical marker 14.

At step 180, position information is disseminated. The positioninformation and associated person 12 information may be displayed on amap, or otherwise communicated. FIGS. 1 and 6 illustrate configurationswhere the position information of the optical markers 14 is displayed ona map on a computer 40 display. Using the position information of thecamera 20 and the position information of the optical marker(s) 14,indicia are displayed on a map at the location corresponding to theassociated position information. Additional information such as theassociated object identifier, timestamp, and position details may alsobe communicated. In other configurations, the position information istransmitted. Such configuration can include wireless transmissions, suchas burst transmission, for example, transmission of optical marker 14position information in the area of the camera(s) 20 field(s) of viewfor display to persons 12 within the area or enabling/disabling weaponsbased on position and/or orientation information.

The camera 20 is monitored for the duration of the session 130.

Insofar as the description above and the accompanying drawing discloseany additional subject matter that is not within the scope of the singleclaim below, the inventions are not dedicated to the public and theright to file one or more applications to claim such additionalinventions is reserved.

What is claimed is:
 1. A method for tracking objects comprising: receiving an optical marker dictionary comprised of one or more optical markers, said optical markers comprising optically distinct indicators associating an optical marker within said optical marker dictionary with an object and affixing said optical marker to said object; a processor in communication with a camera, said processor receiving image data from said camera; said processor applying computer vision to said image data in order to detect the presence of one or more optical markers within said optical marker dictionary within said image data; said processor determining camera position information; said processor applying computer vision to said image data in order to determine relative position information for said detected optical markers; and employing said camera position information and said detected optical markers relative position as input, said processor projecting a position from said camera to said detected optical marker.
 2. The method of claim 1, wherein said optical markers comprises machine-readable optical label formats.
 3. The method of claim 1, wherein said optical markers are outside the visible spectrum.
 4. The method of claim 1, wherein said optical markers are displayed on a print surface.
 5. The method of claim 1, wherein said optical markers are displayed on an electronic visual display.
 6. The method of claim 1, wherein input for said camera position information includes position information from a global navigation system.
 7. The method of claim 1, wherein input for said camera position information includes relative position information from an origin point.
 8. The method of claim 1, wherein input for said camera position information includes relative position information from an anchor node.
 9. The method of claim 1, wherein said image data includes camera information and camera settings.
 10. The method of claim 1, wherein computer vision is employed to determine said relative position.
 11. A system for tracking objects, said system comprising: providing a processor configured to perform the following: receive an optical marker dictionary comprised of one or more optical markers; said optical markers comprising optically distinct indicators to be displayed on an object and associated with said object; said processor in communication with a camera, receiving image data from said camera; said processor applying computer vision to said image data in order to detect the presence of one or more optical markers within said optical marker dictionary within said image data; said processor receiving a camera position information; said processor applying computer vision to said image data in order to determine relative position information for said detected optical markers; and employing said camera position information and said detected optical markers relative position as input, said processor projecting a position from said camera to said detected optical marker.
 12. The system of claim 11, wherein said optical marker generated with target dimensions, and said target dimensions received by said processor.
 13. The system of claim 11, wherein said optical markers are displayed on a fabric surface.
 14. The system of claim 11, wherein computer vision is employed to determine said relative position.
 15. The system of claim 11, wherein laser range finder or lidar input is employed to determine said relative position.
 16. The system of claim 11, wherein said camera position information is determined by time of flight processing from a device with a known location.
 17. An unmanned aerial vehicle for tracking objects, said unmanned aerial vehicle comprising: a processor and camera; said processor configured to perform the following: receive an optical marker dictionary comprised of one or more optical markers; said optical markers comprising optically distinct indicators to be displayed on an object and associated with said object; said processor in communication with said camera, receiving image data from said camera; said processor applying computer vision to said image data in order to detect the presence of one or more optical markers within said optical marker dictionary within said image data; said processor receiving a camera position information; said processor applying computer vision to said image data in order to determine relative position information for said detected optical markers; and employing said camera position information and said detected optical markers relative position as input, said processor projecting a position from said camera to said detected optical marker.
 18. The unmanned aerial vehicle of claim 17, further comprising a global navigation system communicating position information to said processor as input for said camera position.
 19. The unmanned aerial vehicle of claim 17, further comprising a printer, operable to imprint an optical mark to a print surface.
 20. The unmanned aerial vehicle of claim 17, wherein computer vision is employed to determine said relative position. 